Skip to content

fix(executorch): move compile_and_save's engines into the program instead of copying them - #5045

Draft
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:compile-and-save-consume
Draft

shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:compile-and-save-consume

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Depends on pytorch/executorch#25099, and lands after it with the ExecuTorch pin moved, so CI runs the move tests. With an ExecuTorch without consume, such as 1.6.0, compile_and_save copies as before. Other saves are unchanged.

What is wrong today

compile_and_save writes each engine to a temporary file as soon as it is built. The ExecuTorch save copies that file into the .pte (or .ptd), and deletes it only at the end. So every engine is on disk twice, and a large export can fail on a device with room for the result.

What this change does

  • The backend hands each engine file to ExecuTorch as data that owns it (FileBackedData(path, cleanup=True)).
  • Inside compile_and_save, when the outputs would not fit in the free space of their file system (with a tenth to spare), the program and the .ptd are written with consume=True. ExecuTorch then moves each engine into them, so the engines are on disk once, on a file system with sparse files. Other data that owns its file, such as CUDA weights, moves too.
  • When a copy fits, it copies as before. The move syncs each chunk to disk, so it is slower. On btrfs, engines of 0.5, 2 and 8 GiB took 1.3, 8 and 37 seconds to move and 0.5, 1.9 and 6.7 seconds to copy.
  • If a move fails, ExecuTorch can leave <name>.<random>.partial. It can't be finished, so it is now deleted, with any data file the move cut short (such as CUDA weights), and the error says to export again.
  • After the save, each moved engine's plan is emptied, so a later read raises instead of crashing with a bus error.
  • With generate_etrecord, the save copies, because the ETRecord reads the engines afterwards. The docs now say why compile_and_save can write an ETRecord (it keeps each engine as bytes read from its file) and that the record holds copies of every engine.

Tests

@meta-cla meta-cla Bot added the cla signed label Nov 14, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [Python] Issues re: Python API labels Nov 14, 2026
@github-actions
github-actions Bot requested a review from zewenli98 November 14, 2026 22:41
@shoumikhin
shoumikhin force-pushed the compile-and-save-consume branch from 03b81ce to ba01aca Compare November 15, 2026 01:40
@github-actions github-actions Bot added component: core Issues re: The core compiler component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Nov 15, 2026
@shoumikhin
shoumikhin force-pushed the compile-and-save-consume branch 2 times, most recently from 3cf1702 to 306a9de Compare November 15, 2026 07:04
@shoumikhin
shoumikhin marked this pull request as draft November 15, 2026 07:04
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 15, 2026
@shoumikhin
shoumikhin force-pushed the compile-and-save-consume branch 8 times, most recently from 2f360b9 to d53ffa2 Compare November 19, 2026 11:23
…tead of copying them

`compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch
save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the
call returns. So while the program is written, every engine is on disk twice. For a model whose
engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a
device that has room for the result.

The backend now hands each of these engine files to ExecuTorch as data that owns the file
(`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside
`compile_and_save`, when the outputs would not fit in the free space of their file system (with a
tenth to spare, and an output already there counted as free, since both a copy and a move replace
it), the program is then written with ExecuTorch's `save(file_path, consume=True)` and
the `.ptd` with `write_tensor_data_to_file(..., consume=True)`, so ExecuTorch moves each engine into
the program: it copies it back to front and truncates the file as it goes, so the engines are on
disk once (plus one chunk) instead of twice, on a file system with sparse files. ExecuTorch moves
other data that owns its file the same way, such as the CUDA backend's weights, and copies data it
can't move, with a warning. Its `save` takes only a `.pte` path, so a program saved under another
name is written as before, with a copy; the `.ptd` still moves.

ExecuTorch is an optional extra, and releases before pytorch/executorch#25099, such as 1.6.0, have
no `consume` argument. Both writes check whether ExecuTorch's method takes it, through one small
helper, and copy as before when it doesn't. That fallback writes the program through `open` and
`write_to_file`, as before this change, so a failed write still raises.

The move is slower than a copy, because ExecuTorch syncs each chunk to disk before the engine file
gives it up. Measured on btrfs, engines of 0.5, 2 and 8 GiB took 1.3, 8 and 37 seconds to move and
0.5, 1.9 and 6.7 seconds to copy. So `compile_and_save` moves only when a copy would not fit, where
the save failed before, and otherwise copies exactly as before. The docstring says so.

If a move fails after some data left its engine file, ExecuTorch keeps its partial output,
`<name>.<random>.partial`, which holds that data. The engine files are temporary and are deleted
when `compile_and_save` returns, so that output can never be finished, and it can be as large as the
engines on a disk that just ran out. So the consuming writes now run inside `moving_engines_into`:
when the save fails, it deletes each partial output this save left, and a note on the error says to
export again. A partial output that was there before the save is never touched. ExecuTorch can also
leave other data it moved in part, such as a file of CUDA weights whose rest is in the partial
output; that file is deleted too, since nothing can finish it either. Afterwards it empties the plan
of each engine the save moved, in part or in full: that plan maps an emptied file, so a later read
of it through its module, such as a second save in the block, killed the process with a bus error.
Now the plan is empty and that read raises. An engine file the cleanup can't delete is logged as a
warning, so it can't hide the error that ended the block, and a save that succeeded doesn't fail
over it. The program path goes through `os.fspath` first, so a `pathlib.Path` still works.

With `generate_etrecord`, the save copies the engines instead. The ETRecord is written after the
program and reads the engines, so a move would leave it nothing to read. The program is still
written first, as before. The docs now say why `compile_and_save` can write an ETRecord when a
plain save of a program with TensorRT engines can't: it keeps each engine as a byte buffer read
from its file. The record then holds copies of every engine, and writing it reads them into
memory, which the docs also say: for a 4 GiB engine the record took 8 GiB, and the save 3 GiB more
host memory.

The move needs the `consume` argument of `save` and `write_tensor_data_to_file` from
pytorch/executorch#25099; with an ExecuTorch without it, `compile_and_save` copies as before. Only
the move passes `consume`, so every other save calls ExecuTorch exactly as before.

Test plan:

- `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed:
  - New `test_save_in_the_offload_block_moves_the_engines_when_a_copy_does_not_fit`: inside
    `compile_and_save`'s offload block, the program and the `.ptd` are written with
    `consume=True` when the disk has no room for a copy, and without it when it has, counting an
    old output it replaces as free. The path is a `pathlib.Path`.
  - New `test_save_in_the_offload_block_moves_only_the_ptd_into_a_path_without_pte`: the same
    block with a `.bin` path writes the program without `consume` and the `.ptd` with it.
  - New `test_save_in_the_offload_block_copies_with_an_executorch_without_consume`: with an
    ExecuTorch whose writes take no `consume`, the program goes through `write_to_file` and the
    `.ptd` is written without `consume`.
  - New `test_a_failed_move_in_the_offload_block_deletes_its_partial_output`: a consuming `.pte`
    or `.ptd` write that stops inside an engine leaves no partial file, and the error says to
    export again.
  - New `test_a_failed_move_deletes_what_it_moved_in_part`: a failed save deletes the partial output
    it left and a data file it moved in part, and says to export again; an untouched data file and a
    partial output from before the save are kept; the engine files are deleted in every case.
  - New `test_a_save_empties_the_plans_of_the_engines_it_moved`: after a save, a moved engine's
    plan is empty and no longer found as an engine file, the other one is untouched, and a second
    save passes over the moved one.
  - New tests for engine files the cleanup can't check or delete: a warning, not an error.
  - New `test_save_in_the_offload_block_keeps_the_program_when_the_etrecord_fails` and
    `test_save_executorch_does_not_consume`.
- `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100 with TensorRT 11.4, with
  ExecuTorch 1.6 and with ExecuTorch from pytorch/executorch#25099:
  - The streaming test also checks that each engine reaches ExecuTorch as data that owns its file.
  - New `test_engines_move_into_the_program_without_a_second_copy`, new
    `test_an_etrecord_makes_the_save_copy_the_engines`, and new
    `test_a_failed_move_leaves_no_partial_output` (the disk fills after the first engine moved;
    no partial file is left). They make the disk
    look full, so the save moves, and they need an ExecuTorch that moves data. The skip checks
    for `Cord.file_backed_buffers` as well as `consume`, so an ExecuTorch that renames `consume`
    fails them instead of skipping them.

Signed-off-by: Anthony Shoumikhin <anthony@shoumikh.in>
@shoumikhin
shoumikhin force-pushed the compile-and-save-consume branch from d53ffa2 to bc9748e Compare November 20, 2026 13:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant